ci(e2e): run the Playwright suite on pull requests, sharded four ways - #494
Draft
DavidBabinec wants to merge 2 commits into
Draft
ci(e2e): run the Playwright suite on pull requests, sharded four ways#494DavidBabinec wants to merge 2 commits into
DavidBabinec wants to merge 2 commits into
Conversation
The suite was excluded from CI because a serial run took most of an hour and timed out unpredictably. Nothing has gated it since, which is how three specs rotted on main behind security fixes nobody saw break them (#492). Each shard boots its own e2e:dev stack on its own disposable SQLite database, so the shards share no state and the setup projects run once per shard. Measured locally with CI=1: shard 1 of 4 is 68 tests in 6 minutes, against 15.6 minutes for the whole suite serially. It runs as its own workflow so a browser run never delays the build, lint and test feedback a PR needs first. forbidOnly is on under CI, so a stray .only can no longer green the suite by running a single test. Reports and traces upload as artifacts when a shard fails.
The first Actions run failed on every shard the same way: the Bun CMS
came up, Playwright waited on the Vite dev server at :5174, and Vite
printed nothing for two minutes before taking SIGTERM still alive. The
dev server hangs on Linux runners under Bun. A cold start here is 267 ms,
so nothing local could have caught it.
Rather than nurse the dev server through CI, the E2E stack now builds
the admin SPA and serves it from the same Bun process that serves the
published site. No Vite dev server, no second port, no dev-only proxy
plugins in the loop. `vite build` is a batch step and already passes on
Linux CI on every PR; it was only the long-lived dev server that did not.
The suite now exercises the bundle users actually get.
One origin is safe by design: the session cookie is `Path=/admin`, and
`visitPublicPage` opens a fresh browser context, so anonymity never
depended on the port. The public specs were already hitting Bun on :3002
directly; only the admin moves, from Vite-served to dist-served.
`scripts/e2e-dev.ts` is replaced by `scripts/e2e-server.ts`
(`bun run e2e:serve`), the Playwright `webServer` budget covers the
build, `tests/e2e/helpers/constants.ts` exports one `ADMIN_BASE_URL`
that `account.e2e.ts` now imports instead of duplicating, and the
`devWorkflow` gate test asserts the new shape.
Running the whole suite in one process for the first time also surfaced
a shared-state leak that sharding had hidden: `content-seo-meta` publishes
a Posts template at priority 300 and never removes it, and the template
chain keeps one template per breadth level, so CONTENT-008 (priority 150)
and SITE-018 (a 300 that loses the tie on document order) rendered through
the wrong template whenever that spec ran first. A shared `deleteTemplate`
helper now removes every Posts template a spec publishes before the spec
ends, in all five places one is created.
Verification (local, Chromium, new harness):
bunx playwright test --project=setup 1 passed
content-seo-meta + content + visual-builder 38 passed (the leak ordering)
bunx playwright test --shard=1/4 68 passed
bunx playwright test --shard=2/4 29 passed (was 1 failed before the cleanup)
bunx playwright test --shard=3/4 37 passed
bunx playwright test --shard=4/4 44 passed
full suite, one process 161 passed, 1 failed, 1 flaky
failed: CONTENT-009, the client-side untitled-slug pick (product bug,
tracked separately); flaky: SITE-005 drag (timing, passes on retry)
bunx eslint tests/e2e/… clean
bunx tsc -p tests/e2e/tsconfig.json --noEmit clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Runs the Playwright suite on every pull request, sharded four ways, against the built admin SPA served by the Bun server on one origin.
.github/workflows/e2e.yml: four-shard matrix,fail-fast: false, report artifacts on failure.scripts/e2e-server.tsreplacesscripts/e2e-dev.ts(bun run e2e:serve): wipe.tmp/e2e-*,bun run build, then servedist/and the public site fromserver/index.tson127.0.0.1:3002. No Vite dev server, no second port, no dev-only proxy plugins in the loop.playwright.config.ts: one base URL for admin and public; thewebServerbudget covers the build;forbidOnlyunder CI.tests/e2e/helpers/constants.tsexportsADMIN_BASE_URL;account.e2e.tsimports it instead of duplicating the literal.tests/e2e/helpers/editor.tsgainsdeleteTemplate. Every spec that publishes a Posts template now removes it before it ends (five sites acrosscontent-seo-meta,content,visual-builder).docs/e2e/README.mddocuments the stack and the new Template rule; thedevWorkflowgate test asserts the new shape.Why
The first Actions run failed on all four shards identically: the CMS came up, Playwright waited on the Vite dev server at
:5174, and Vite printed nothing for two minutes before taking SIGTERM still alive. The Vite dev server hangs on Linux runners under Bun. A cold start here is 267 ms, so nothing local could catch it.vite buildis a batch step and already passes on Linux CI on every PR; only the long-lived dev server is the problem, so the suite stops depending on it. It now exercises the bundle users actually get.One origin is safe by design: the session cookie is
Path=/admin(server/handlers/cms/session.ts) andvisitPublicPageopens a fresh browser context, so anonymity never depended on the port. Public specs were already hitting Bun on:3002; only the admin moves, from Vite-served todist-served.Running the suite in one process for the first time also surfaced a shared-database leak that sharding had hidden.
content-seo-metapublished a Posts template at priority 300 and never removed it, and the template chain keeps one template per breadth level (highest priority, then document order), so CONTENT-008 (priority 150) and SITE-018 (a 300 that loses the tie) rendered through the wrong template whenever it ran first. Fixed at the source with cleanup, not by quarantining.Impact
bunx playwright testnow builds first (about 12 s locally).E2E_REUSE_SERVER=1plusE2E_ADMIN_BASE_URLstill points the suite at a server you started yourself.createUntitledEntrypicks the new slug from the client-sideentries.length, and an earlier spec leaves an unsaveduntitledrow behind. That is a product bug (a real user with a slow list load gets "Could not create entry"). It passes under sharding, which is why CI is green and the bug is still real.Verification
Local, Chromium, new harness, real exit codes:
--project=setupcontent-seo-meta,content,visual-builder(the leak ordering)--shard=1/4--shard=2/4--shard=3/4--shard=4/4bunx eslint tests/e2e/…andbunx tsc -p tests/e2e/tsconfig.json --noEmit